Image Segmentation and Scene Understanding Project

1. Introduction Scene or image understanding deals with the problem of making a computer " understand " the world behind the image. This can be done in a number of different ways. In this project, we will deal with a kind of problem of scene understanding, semantic image segmentation or pixel labeling. Multi-class image segmentation or pixel labeling does more than the task of object recognition, which just recognizes a particular kind of object. It tries to do object recognition and classify all pixels in an image to particular object classes concurrently. That is, we take an image in and then automatically breaks the image into regions and labels each region with a semantic class label. For example, when the computer reads an image of a person riding a horse on the grass, it can segment the image into several regions with semantic class labels such as person, horse, grass or even sky, etc. (In our project, this is implemented by semantic pixel labeling.) In this semester, my work on the project was divided into 2 phases. In the first phase, I developed a web application on scene understanding where users can easily submit images to the website server and do the state-of-the-art scene understanding work on their submitted images. The server processes the images using a current state-of-the-art scene understanding model and return image segmentation results to the user. Importantly, the web application will maintain a record of all previously processed images so that it can avoid recalculating results on these images (and simply return the results to the user). Also, it can be easily extended to include more and more scene understanding models to make it much more convenient to compare the segmentation results of different models. In the second phase, I extended the current state of the art scene understanding models by incorporating relative location information. This is built on recent work on scene understanding (e.g., [2] and [1]). [1] describes a method of incorporating relative location information of different classes as local features on the superpixels level, and I applied this method on the pixels level. That is, to overcome the challenge of computation on such global information, [1] proposed an approximate two-stage processing (learning and prediction). In the first stage, we still use the current framework and current local features to train the model to get predictions on the pixels' labels. Then, we use the …